{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Our Plan\n",
        "We want to further pre-train the bert-based-uncased model using a clinical notes dataset. Here is our plan:\n",
        "\n",
        "1. Import necessary packages.\n",
        "2. Download the dataset from Kaggle.\n",
        "  * We will use the akashadesai/clinical-notes dataset from Kaggle.\n",
        "  * Save the dataset on your machine.\n",
        "3. Preprocess the dataset.\n",
        "  * Create a pandas dataframe with each sentence in a new row.\n",
        "  * Ensure consecutive sentences are in consecutive rows (e.g., Sentence A in row i and Sentence B in row i+1).\n",
        "3. Create a custom Dataset class.\n",
        "  * For BERT training, each item should be in the format: `Sentence A + [SEP] + Sentence B`.\n",
        "  * The __getitem__ method should return the tokenization of (`Sentence A + [SEP] + Sentence B`).\n",
        "4. Create a DataCollatorForPreTraining class. This will be passed as the collate_fn in the DataLoader.\n",
        "  * The class should inherit from DataCollatorForLanguageModeling.\n",
        "  * Mask a few tokens from Sentence A.\n",
        "5. Create a DataLoader.\n",
        "6. Declare the model, loss, and optimizer.\n",
        "7. Prepare the accelerator for GPU and distributed training.\n",
        "8. Perform further training of the bert-based-uncased model.\n",
        "The Clinical BERT model is now ready.\n",
        "\n"
      ],
      "metadata": {
        "id": "xxvwNbX8PJpl"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 1. Import Necessary Packages"
      ],
      "metadata": {
        "id": "lA4XlD7BY1hQ"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "id": "iWFzwrpLo4t4"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import pandas as pd\n",
        "from torch.utils.data import DataLoader, Dataset\n",
        "from transformers import BartTokenizer, BartForConditionalGeneration\n",
        "from transformers import DataCollatorForSeq2Seq\n",
        "from accelerate import Accelerator\n",
        "\n",
        "accelerator = Accelerator()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 2. Download the Dataset from Kaggle\n",
        "Please Download dataset from [https://www.kaggle.com/datasets/akashadesai/clinical-notes](https://www.kaggle.com/datasets/akashadesai/clinical-notes)"
      ],
      "metadata": {
        "id": "GqmvPO7DOuXm"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "5Xhru1PAOrDf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#3. Pre-process the Dataset\n",
        "The following function takes a dataframe as an argument. The dataframe contains a column named 'TEXT' which consists of clinical notes from different patients. In this function:\n",
        "\n",
        "* We remove special characters from the clinical notes.\n",
        "* We split the clinical notes into individual sentences.\n",
        "* We create a new dataframe, where each row holds a single sentence."
      ],
      "metadata": {
        "id": "5wI2mMbSZGS0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import nltk\n",
        "import re\n",
        "from nltk.tokenize import sent_tokenize\n",
        "\n",
        "def create_sentence_dataframe(df):\n",
        "    # Initialize an empty list to store sentences\n",
        "    sentences = []\n",
        "\n",
        "    # Define a pattern to match special characters\n",
        "    special_chars_pattern = re.compile(r'[^a-zA-Z0-9\\s.,?!]+|\\n')\n",
        "    # Loop through each row in the DataFrame\n",
        "    for text in df['TEXT']:\n",
        "        # Remove special characters from the text\n",
        "        clean_text = special_chars_pattern.sub('', text)\n",
        "\n",
        "        # Tokenize the cleaned text into sentences\n",
        "        tokenized_sentences = sent_tokenize(clean_text)\n",
        "\n",
        "        # Add the tokenized sentences to the list\n",
        "        sentences.extend(tokenized_sentences)\n",
        "\n",
        "    # Create a new DataFrame with the sentences\n",
        "    sentence_df = pd.DataFrame(sentences, columns=['text'])\n",
        "\n",
        "    return sentence_df"
      ],
      "metadata": {
        "id": "kb51QJr5m9Bg"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "data_txt = pd.read_csv(\"/Users/premtimsina/Documents/bpbbook/chapter5/dataset/medical_data.csv\")"
      ],
      "metadata": {
        "id": "ntUXw8wMp6T3"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "pd.options.display.max_colwidth = 100\n",
        "data=create_sentence_dataframe(data_txt)"
      ],
      "metadata": {
        "id": "QJ5zoKgmnJS8"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "data.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "YDbnS0y-qEAC",
        "outputId": "701698ea-20a3-4013-d6cc-484ff9660ade"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                                                                  text\n",
              "0  Admission Date  216233              Discharge Date   2162325Date of Birth  208014             Se...\n",
              "1        Known lastname 1829 was seen at Hospital1 18 after a mechanical fall froma height of 10 feet.\n",
              "2     CT scan noted unstable fracture of C67 posterior elements.Major Surgical or Invasive Procedure1.\n",
              "3  Anterior cervical osteotomy, C6C7, with decompression andexcision of ossification of the posteri...\n",
              "4                                                            Anterior cervical deformity correction.3."
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Admission Date  216233              Discharge Date   2162325Date of Birth  208014             Se...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Known lastname 1829 was seen at Hospital1 18 after a mechanical fall froma height of 10 feet.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>CT scan noted unstable fracture of C67 posterior elements.Major Surgical or Invasive Procedure1.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Anterior cervical osteotomy, C6C7, with decompression andexcision of ossification of the posteri...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Anterior cervical deformity correction.3.</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#4. Create a custom Dataset class.\n",
        "* For BERT training, each item should be in the format: Sentence A + [SEP] + Sentence B.\n",
        "* The getitem method should return the tokenization of (Sentence A + [SEP] + Sentence B)."
      ],
      "metadata": {
        "id": "ZJ684tNVaDKs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import BertTokenizer\n",
        "\n",
        "class ClinicalDataset(Dataset):\n",
        "    def __init__(self, data, tokenizer, max_length=512):\n",
        "        self.data = data\n",
        "        self.tokenizer = tokenizer\n",
        "        self.max_length = max_length\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.data)\n",
        "\n",
        "    def __getitem__(self, idx):\n",
        "        news = self.data.loc[idx, \"text\"]\n",
        "        if idx + 1 < len(self.data):\n",
        "            next_news = self.data.loc[idx + 1, \"text\"]\n",
        "        else:\n",
        "            next_news = self.data.loc[0, \"text\"]\n",
        "\n",
        "        combined_news = news + \" [SEP] \" + next_news\n",
        "        tokenized = self.tokenizer(combined_news, truncation=True, padding=\"max_length\", max_length=self.max_length, return_tensors=\"pt\")\n",
        "        return {\"input_ids\": tokenized[\"input_ids\"].squeeze(0), \"attention_mask\": tokenized[\"attention_mask\"].squeeze(0), \"text\": combined_news}\n"
      ],
      "metadata": {
        "id": "u4MVzPSdqH4s"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer = BertTokenizer.from_pretrained(\"bert-base-uncased\")\n",
        "\n",
        "dataset=ClinicalDataset(data,tokenizer)\n",
        "dataset[0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "y8yvyCS4xHn1",
        "outputId": "69eb7c24-7e3f-4fe4-9d47-58a780c998f2"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'input_ids': tensor([  101,  9634,  3058, 20294, 21926,  2509, 11889,  3058, 20294, 21926,\n",
              "         17788, 13701,  1997,  4182, 18512, 24096,  2549,  3348,  5796,  2121,\n",
              "          7903,  2063,  4200, 24164, 10623,  3111, 24343,  2680,  2004,  2383,\n",
              "          2053,  2124,  2035,  2121, 17252,  2000,  5850, 19321, 18537,  8873,\n",
              "         12096,  2171,  2509,  1048,  2546, 11517,  5428, 12879, 12087,  2213,\n",
              "          2099,  1012,   102,  2124,  2197, 18442, 11523,  2001,  2464,  2012,\n",
              "          2902,  2487,  2324,  2044,  1037,  6228,  2991,  2013,  2050,  4578,\n",
              "          1997,  2184,  2519,  1012,   102,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
              "             0,     0]),\n",
              " 'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
              "         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
              "         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
              "         1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
              "         0, 0, 0, 0, 0, 0, 0, 0]),\n",
              " 'text': 'Admission Date  216233              Discharge Date   2162325Date of Birth  208014             Sex   MService MEDICINEAllergiesPatient recorded as having No Known Allergies to DrugsAttendingFirst Name3 LF 1828Chief ComplaintMr. [SEP] Known lastname 1829 was seen at Hospital1 18 after a mechanical fall froma height of 10 feet.'}"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer.sep_token_id"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vJrjm2mwpMNL",
        "outputId": "b48d39f2-f62e-4eaa-aa2f-7d1a436229df"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "102"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 5. Create a DataCollatorForPreTraining class\n",
        "This is what we are performing in the `DataCollatorForPreTraining` class:\n",
        "1. We inherit from the DataCollatorForLanguageModeling class provided by the Hugging Face Transformers library. We will use `DataCollatorForLanguageModeling` for the MLM.\n",
        "2. We override the __call__ method to process the input examples for the pre-training task.\n",
        "  * We initialize lists to store NSP labels, input IDs, attention masks, and labels for each example.\n",
        "  * We aim to create 50% sentence pairs as NSP (Next Sentence Prediction) True and 50% sentence pairs as NSP False.\n",
        "  * In the following function, if random.random() > 0.5, we consider the sentence pair as a True NSP pair. Since the data coming from the Dataset class is already a True NSP pair, we don't need to modify it.\n",
        "  * On the other hand, if random.random() < 0.5, we consider the sentence pair as a False NSP pair. To achieve this, we shuffle the tokens in the next sentence. As a result, the sentence after [SEP] is not the true next sentence, making it an NSP False pair.\n",
        "3. We use the parent class's __call__ method to handle the MLM task for the examples.\n",
        "4. We add NSP labels to the batch and return the final batch for further processing in the pre-training loop."
      ],
      "metadata": {
        "id": "MaTw0obTbYwn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import DataCollatorForLanguageModeling\n",
        "import random\n",
        "\n",
        "\n",
        "class DataCollatorForPreTraining(DataCollatorForLanguageModeling):\n",
        "    def __init__(self, tokenizer, mlm=True, mlm_probability=0.15, nsp_probability=0.5):\n",
        "        super().__init__(tokenizer=tokenizer, mlm=mlm, mlm_probability=mlm_probability)\n",
        "        self.nsp_probability = nsp_probability\n",
        "\n",
        "    def __call__(self, examples):\n",
        "        # NSP labels\n",
        "        nsp_labels = []\n",
        "\n",
        "        input_ids_list = []\n",
        "        attention_masks_list = []\n",
        "        labels_list = []\n",
        "\n",
        "        # Create NSP input\n",
        "        for example in examples:\n",
        "            input_ids = example[\"input_ids\"]\n",
        "            attention_mask = example[\"attention_mask\"]\n",
        "\n",
        "            if random.random() > self.nsp_probability:\n",
        "                # Is Next Sentence\n",
        "                nsp_labels.append(1)\n",
        "            else:\n",
        "                # Not Next Sentence\n",
        "                nsp_labels.append(0)\n",
        "\n",
        "                # Shuffle second sentence\n",
        "                sep_idx = (input_ids == self.tokenizer.sep_token_id).nonzero(as_tuple=True)[0][0].item()\n",
        "                second_sentence = input_ids[sep_idx + 1:]\n",
        "                second_sentence = second_sentence[torch.randperm(second_sentence.size()[0])]\n",
        "\n",
        "                # Concatenate first sentence and shuffled second sentence\n",
        "                input_ids = torch.cat((input_ids[:sep_idx + 1], second_sentence), dim=0)\n",
        "\n",
        "            input_ids_list.append(input_ids)\n",
        "            attention_masks_list.append(attention_mask)\n",
        "\n",
        "            # Mask only the first sentence\n",
        "            sep_idx = (input_ids == self.tokenizer.sep_token_id).nonzero(as_tuple=True)[0][0].item()\n",
        "            labels = input_ids.clone()\n",
        "            labels[sep_idx:] = -100\n",
        "            labels_list.append(labels)\n",
        "\n",
        "        # Create a list of dictionaries for the parent class\n",
        "        example_dicts = [{\"input_ids\": ids, \"attention_mask\": mask, \"labels\": lbl} for ids, mask, lbl in zip(input_ids_list, attention_masks_list, labels_list)]\n",
        "        \n",
        "        # Handle MLM using the parent class\n",
        "        batch = super().__call__(example_dicts)\n",
        "\n",
        "        # Add NSP labels to batch\n",
        "        batch[\"next_sentence_label\"] = torch.tensor(nsp_labels, dtype=torch.long)\n",
        "\n",
        "        return batch\n"
      ],
      "metadata": {
        "id": "GAErEYgzqoyf"
      },
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 6. Create DataLoader\n",
        "Let's Discuss each item of dataloader.\n",
        "1. input_ids: It is token number of each token. The padded token has the token number of 0. \n",
        "2. attention_mask: 1 signifies true token, 0 signifies padded token\n",
        "3. labels: It is the label for Masked Language Modeling task.\n",
        " * `-100` means do not use that token to calculate loss function; or, the correspoding token will not be masked\n",
        " * non `-100` signifies that the corresponding token will be masked and use for MLM pre-training objective.\n",
        "\n"
      ],
      "metadata": {
        "id": "kwpZY7VJe6eQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from torch.utils.data import DataLoader\n",
        "\n",
        "# Instantiate the tokenizer, dataset, and data collator\n",
        "data_collator = DataCollatorForPreTraining(tokenizer)\n",
        "\n",
        "# Create the DataLoader\n",
        "train_dataloader = DataLoader(\n",
        "    dataset, shuffle=True, collate_fn=data_collator, batch_size=16\n",
        ")"
      ],
      "metadata": {
        "id": "jTr4YhD-5s1b"
      },
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 6.1 We are just reviewing what dataloader looks like."
      ],
      "metadata": {
        "id": "5XswTHUN2ZjB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "item=next(iter(train_dataloader))\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "QYTPzWhlq0Oc"
      },
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(len(train_dataloader))\n",
        "print('ids', item['input_ids'][0])\n",
        "print('mask', item['attention_mask'][0])\n",
        "print('labels', item['labels'][0])\n",
        "print('next_sentence_label',item['next_sentence_label'][0])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NdmccWuaFztq",
        "outputId": "2a3328ec-55cd-475b-b50e-79d7bdd5643a"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3474\n",
            "ids tensor([  101,   103,   103,  3298,  2138,  1997,  2010,  3532, 17084,  1012,\n",
            "          102,  2196, 20482,  1012,   102,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0])\n",
            "mask tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
            "        0, 0, 0, 0, 0, 0, 0, 0])\n",
            "labels tensor([-100, 2515, 2025, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,\n",
            "        -100, -100, -100, -100, -100, -100, -100, -100])\n",
            "next_sentence_label tensor(1)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#7. Pre-training\n",
        "1. `model = BertForPreTraining(config)`: Here is the reason why we are creating instance of `BertForPreTraining`\n",
        "  * It is specifically designed for pre-training the BERT model architecture. The class encapsulates the BERT architecture along with additional pre-training tasks: NSP and MLM\n",
        "  * This is a crucial consideration: if you cannot find a module that satisfies the pre-training objective of a particular model, you will need to create the module yourself. In our case, Hugging Face's BertForPreTraining module already met both NSP and MLM pre-training objectives, so we didn't need to write a custom module.\n",
        "  * At the time of writing this book, I could not find a BartForPretraining module that satisfied BART's pre-training objectives. Therefore, if we want to pre-train BART, we would need to create a custom module for further pre-training BART.\n",
        "\n",
        "2. In the following code, we are using just one epoch. However, to achieve optimal results, you should consider using multiple epochs."
      ],
      "metadata": {
        "id": "sGM8C80ofKgC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "from torch.nn import CrossEntropyLoss\n",
        "from torch.optim import AdamW\n",
        "from transformers import BertForPreTraining, BertConfig\n",
        "from accelerate import Accelerator\n",
        "\n",
        "# Initialize accelerator\n",
        "accelerator = Accelerator()\n",
        "\n",
        "# Load BERT model\n",
        "config = BertConfig.from_pretrained(\"bert-base-uncased\")\n",
        "model = BertForPreTraining(config)\n",
        "\n",
        "# Set up the optimizer\n",
        "optimizer = AdamW(model.parameters(), lr=5e-5)\n",
        "\n",
        "# Prepare the model and optimizer for acceleration\n",
        "model, optimizer, train_dataloader = accelerator.prepare(model, optimizer, train_dataloader)\n",
        "\n",
        "# Set training parameters\n",
        "num_epochs = 1\n",
        "print_every = 10\n",
        "\n",
        "# Training loop\n",
        "for epoch in range(num_epochs):\n",
        "    print(f\"Epoch {epoch + 1}/{num_epochs}\")\n",
        "    model.train()\n",
        "    running_loss = 0.0\n",
        "\n",
        "    for step, batch in enumerate(train_dataloader):\n",
        "        input_ids = batch[\"input_ids\"].to(accelerator.device)\n",
        "        attention_mask = batch[\"attention_mask\"].to(accelerator.device)\n",
        "        labels = batch[\"labels\"].to(accelerator.device)\n",
        "        next_sentence_label = batch[\"next_sentence_label\"].to(accelerator.device)\n",
        "\n",
        "        # Forward pass\n",
        "        outputs = model(input_ids=input_ids, attention_mask=attention_mask, labels=labels, next_sentence_label=next_sentence_label)\n",
        "        loss = outputs.loss\n",
        "\n",
        "        # Backward pass\n",
        "        accelerator.backward(loss)\n",
        "        optimizer.step()\n",
        "        optimizer.zero_grad()\n",
        "\n",
        "        running_loss += loss.item()\n",
        "\n",
        "        if (step + 1) % print_every == 0:\n",
        "            print(f\"Step {step + 1}: Loss = {running_loss / print_every:.4f}\")\n",
        "            running_loss = 0.0\n",
        "\n",
        "print(\"Training complete!\")\n",
        "\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7BRhsG09YXK5",
        "outputId": "f6e0df8b-5b47-41bf-fbbe-97cf2494b0c0"
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/1\n",
            "Step 10: Loss = 1.0446\n",
            "Step 20: Loss = 0.9181\n",
            "Step 30: Loss = 0.9136\n",
            "Step 40: Loss = 0.9404\n",
            "Step 50: Loss = 0.8646\n",
            "Step 60: Loss = 0.8865\n",
            "Step 70: Loss = 0.9166\n",
            "Step 80: Loss = 0.8685\n",
            "Step 90: Loss = 0.8547\n",
            "Step 100: Loss = 0.8541\n",
            "Step 110: Loss = 0.8169\n",
            "Step 120: Loss = 0.8246\n",
            "Step 130: Loss = 0.8463\n",
            "Step 140: Loss = 0.8528\n",
            "Step 150: Loss = 0.7851\n",
            "Step 160: Loss = 0.7743\n",
            "Step 170: Loss = 0.6331\n",
            "Step 180: Loss = 0.4951\n",
            "Step 190: Loss = 0.7283\n",
            "Step 200: Loss = 1.0740\n",
            "Step 210: Loss = 0.9100\n",
            "Step 220: Loss = 0.8492\n",
            "Step 230: Loss = 0.8410\n",
            "Step 240: Loss = 0.8498\n",
            "Step 250: Loss = 0.8160\n",
            "Step 260: Loss = 0.8486\n",
            "Step 270: Loss = 0.8349\n",
            "Step 280: Loss = 0.8328\n",
            "Step 290: Loss = 0.8725\n",
            "Step 300: Loss = 0.8514\n",
            "Step 310: Loss = 0.8503\n",
            "Step 320: Loss = 0.8676\n",
            "Step 330: Loss = 0.8211\n",
            "Step 340: Loss = 0.8593\n",
            "Step 350: Loss = 0.8815\n",
            "Step 360: Loss = 0.8553\n",
            "Step 370: Loss = 0.8196\n",
            "Step 380: Loss = 0.8281\n",
            "Step 390: Loss = 0.8457\n",
            "Step 400: Loss = 0.8193\n",
            "Step 410: Loss = 0.8452\n",
            "Step 420: Loss = 0.8292\n",
            "Step 430: Loss = 0.8553\n",
            "Step 440: Loss = 0.8596\n",
            "Step 450: Loss = 0.8505\n",
            "Step 460: Loss = 0.8559\n",
            "Step 470: Loss = 0.8478\n",
            "Step 480: Loss = 0.8429\n",
            "Step 490: Loss = 0.8519\n",
            "Step 500: Loss = 0.8547\n",
            "Step 510: Loss = 0.8419\n",
            "Step 520: Loss = 0.8572\n",
            "Step 530: Loss = 0.8658\n",
            "Step 540: Loss = 0.8160\n",
            "Step 550: Loss = 0.8197\n",
            "Step 560: Loss = 0.8581\n",
            "Step 570: Loss = 0.8592\n",
            "Step 580: Loss = 0.8365\n",
            "Step 590: Loss = 0.8729\n",
            "Step 600: Loss = 0.8582\n",
            "Step 610: Loss = 0.8391\n",
            "Step 620: Loss = 0.8535\n",
            "Step 630: Loss = 0.8479\n",
            "Step 640: Loss = 0.8441\n",
            "Step 650: Loss = 0.8305\n",
            "Step 660: Loss = 0.8162\n",
            "Step 670: Loss = 0.8586\n",
            "Step 680: Loss = 0.8542\n",
            "Step 690: Loss = 0.8424\n",
            "Step 700: Loss = 0.8442\n",
            "Step 710: Loss = 0.8605\n",
            "Step 720: Loss = 0.8293\n",
            "Step 730: Loss = 0.8317\n",
            "Step 740: Loss = 0.8315\n",
            "Step 750: Loss = 0.8379\n",
            "Step 760: Loss = 0.8431\n",
            "Step 770: Loss = 0.8228\n",
            "Step 780: Loss = 0.8358\n",
            "Step 790: Loss = 0.8425\n",
            "Step 800: Loss = 0.8866\n",
            "Step 810: Loss = 0.8469\n",
            "Step 820: Loss = 0.8735\n",
            "Step 830: Loss = 0.8643\n",
            "Step 840: Loss = 0.8266\n",
            "Step 850: Loss = 0.8284\n",
            "Step 860: Loss = 0.8487\n",
            "Step 870: Loss = 0.8316\n",
            "Step 880: Loss = 0.8355\n",
            "Step 890: Loss = 0.8592\n",
            "Step 900: Loss = 0.8492\n",
            "Step 910: Loss = 0.8340\n",
            "Step 920: Loss = 0.8386\n",
            "Step 930: Loss = 0.8090\n",
            "Step 940: Loss = 0.7144\n",
            "Step 950: Loss = 0.8410\n",
            "Step 960: Loss = 0.8296\n",
            "Step 970: Loss = 0.9266\n",
            "Step 980: Loss = 0.6817\n",
            "Step 990: Loss = 0.3086\n",
            "Step 1000: Loss = 0.2291\n",
            "Step 1010: Loss = 0.5625\n",
            "Step 1020: Loss = 0.5084\n",
            "Step 1030: Loss = 0.8870\n",
            "Step 1040: Loss = 0.7995\n",
            "Step 1050: Loss = 0.7346\n",
            "Step 1060: Loss = 0.3332\n",
            "Step 1070: Loss = 0.1366\n",
            "Step 1080: Loss = 0.4788\n",
            "Step 1090: Loss = 1.1464\n",
            "Step 1100: Loss = 0.8686\n",
            "Step 1110: Loss = 0.8628\n",
            "Step 1120: Loss = 0.8594\n",
            "Step 1130: Loss = 0.8943\n",
            "Step 1140: Loss = 0.8552\n",
            "Step 1150: Loss = 0.8432\n",
            "Step 1160: Loss = 0.8319\n",
            "Step 1170: Loss = 0.8508\n",
            "Step 1180: Loss = 0.8440\n",
            "Step 1190: Loss = 0.8355\n",
            "Step 1200: Loss = 0.8458\n",
            "Step 1210: Loss = 0.8469\n",
            "Step 1220: Loss = 0.8406\n",
            "Step 1230: Loss = 0.8536\n",
            "Step 1240: Loss = 0.8540\n",
            "Step 1250: Loss = 0.8510\n",
            "Step 1260: Loss = 0.8523\n",
            "Step 1270: Loss = 0.8426\n",
            "Step 1280: Loss = 0.8413\n",
            "Step 1290: Loss = 0.8452\n",
            "Step 1300: Loss = 0.8380\n",
            "Step 1310: Loss = 0.8268\n",
            "Step 1320: Loss = 0.8573\n",
            "Step 1330: Loss = 0.8379\n",
            "Step 1340: Loss = 0.8342\n",
            "Step 1350: Loss = 0.8381\n",
            "Step 1360: Loss = 0.8339\n",
            "Step 1370: Loss = 0.8671\n",
            "Step 1380: Loss = 0.8643\n",
            "Step 1390: Loss = 0.8357\n",
            "Step 1400: Loss = 0.8245\n",
            "Step 1410: Loss = 0.8860\n",
            "Step 1420: Loss = 0.8306\n",
            "Step 1430: Loss = 0.8599\n",
            "Step 1440: Loss = 0.8113\n",
            "Step 1450: Loss = 0.8351\n",
            "Step 1460: Loss = 0.8295\n",
            "Step 1470: Loss = 0.8578\n",
            "Step 1480: Loss = 0.8514\n",
            "Step 1490: Loss = 0.8199\n",
            "Step 1500: Loss = 0.8841\n",
            "Step 1510: Loss = 0.8871\n",
            "Step 1520: Loss = 0.8697\n",
            "Step 1530: Loss = 0.8533\n",
            "Step 1540: Loss = 0.8672\n",
            "Step 1550: Loss = 0.8380\n",
            "Step 1560: Loss = 0.8243\n",
            "Step 1570: Loss = 0.8346\n",
            "Step 1580: Loss = 0.8280\n",
            "Step 1590: Loss = 0.8383\n",
            "Step 1600: Loss = 0.8556\n",
            "Step 1610: Loss = 0.8477\n",
            "Step 1620: Loss = 0.8148\n",
            "Step 1630: Loss = 0.8593\n",
            "Step 1640: Loss = 0.8491\n",
            "Step 1650: Loss = 0.8528\n",
            "Step 1660: Loss = 0.8627\n",
            "Step 1670: Loss = 0.8571\n",
            "Step 1680: Loss = 0.8538\n",
            "Step 1690: Loss = 0.8261\n",
            "Step 1700: Loss = 0.8394\n",
            "Step 1710: Loss = 0.8370\n",
            "Step 1720: Loss = 0.8602\n",
            "Step 1730: Loss = 0.8497\n",
            "Step 1740: Loss = 0.8327\n",
            "Step 1750: Loss = 0.8640\n",
            "Step 1760: Loss = 0.8484\n",
            "Step 1770: Loss = 0.8250\n",
            "Step 1780: Loss = 0.8325\n",
            "Step 1790: Loss = 0.8530\n",
            "Step 1800: Loss = 0.8570\n",
            "Step 1810: Loss = 0.8365\n",
            "Step 1820: Loss = 0.8257\n",
            "Step 1830: Loss = 0.8505\n",
            "Step 1840: Loss = 0.8793\n",
            "Step 1850: Loss = 0.8283\n",
            "Step 1860: Loss = 0.8662\n",
            "Step 1870: Loss = 0.8210\n",
            "Step 1880: Loss = 0.8513\n",
            "Step 1890: Loss = 0.8561\n",
            "Step 1900: Loss = 0.8159\n",
            "Step 1910: Loss = 0.8402\n",
            "Step 1920: Loss = 0.8485\n",
            "Step 1930: Loss = 0.8287\n",
            "Step 1940: Loss = 0.8210\n",
            "Step 1950: Loss = 0.8525\n",
            "Step 1960: Loss = 0.8631\n",
            "Step 1970: Loss = 0.8512\n",
            "Step 1980: Loss = 0.8588\n",
            "Step 1990: Loss = 0.8424\n",
            "Step 2000: Loss = 0.8323\n",
            "Step 2010: Loss = 0.8542\n",
            "Step 2020: Loss = 0.8475\n",
            "Step 2030: Loss = 0.8271\n",
            "Step 2040: Loss = 0.8255\n",
            "Step 2050: Loss = 0.8242\n",
            "Step 2060: Loss = 0.8537\n",
            "Step 2070: Loss = 0.8308\n",
            "Step 2080: Loss = 0.8338\n",
            "Step 2090: Loss = 0.8291\n",
            "Step 2100: Loss = 0.8314\n",
            "Step 2110: Loss = 0.8815\n",
            "Step 2120: Loss = 0.8438\n",
            "Step 2130: Loss = 0.8458\n",
            "Step 2140: Loss = 0.8484\n",
            "Step 2150: Loss = 0.8442\n",
            "Step 2160: Loss = 0.8386\n",
            "Step 2170: Loss = 0.8386\n",
            "Step 2180: Loss = 0.8507\n",
            "Step 2190: Loss = 0.8399\n",
            "Step 2200: Loss = 0.8179\n",
            "Step 2210: Loss = 0.8239\n",
            "Step 2220: Loss = 0.8334\n",
            "Step 2230: Loss = 0.8346\n",
            "Step 2240: Loss = 0.8437\n",
            "Step 2250: Loss = 0.8348\n",
            "Step 2260: Loss = 0.8594\n",
            "Step 2270: Loss = 0.8426\n",
            "Step 2280: Loss = 0.8787\n",
            "Step 2290: Loss = 0.8317\n",
            "Step 2300: Loss = 0.8471\n",
            "Step 2310: Loss = 0.8409\n",
            "Step 2320: Loss = 0.8134\n",
            "Step 2330: Loss = 0.8389\n",
            "Step 2340: Loss = 0.8567\n",
            "Step 2350: Loss = 0.8194\n",
            "Step 2360: Loss = 0.8188\n",
            "Step 2370: Loss = 0.8589\n",
            "Step 2380: Loss = 0.8506\n",
            "Step 2390: Loss = 0.8587\n",
            "Step 2400: Loss = 0.8383\n",
            "Step 2410: Loss = 0.8581\n",
            "Step 2420: Loss = 0.8595\n",
            "Step 2430: Loss = 0.8427\n",
            "Step 2440: Loss = 0.8386\n",
            "Step 2450: Loss = 0.8413\n",
            "Step 2460: Loss = 0.8314\n",
            "Step 2470: Loss = 0.8383\n",
            "Step 2480: Loss = 0.8765\n",
            "Step 2490: Loss = 0.8562\n",
            "Step 2500: Loss = 0.8331\n",
            "Step 2510: Loss = 0.8451\n",
            "Step 2520: Loss = 0.8431\n",
            "Step 2530: Loss = 0.8460\n",
            "Step 2540: Loss = 0.8284\n",
            "Step 2550: Loss = 0.8589\n",
            "Step 2560: Loss = 0.8210\n",
            "Step 2570: Loss = 0.8352\n",
            "Step 2580: Loss = 0.8326\n",
            "Step 2590: Loss = 0.8560\n",
            "Step 2600: Loss = 0.8384\n",
            "Step 2610: Loss = 0.8553\n",
            "Step 2620: Loss = 0.8657\n",
            "Step 2630: Loss = 0.8638\n",
            "Step 2640: Loss = 0.8371\n",
            "Step 2650: Loss = 0.8292\n",
            "Step 2660: Loss = 0.8483\n",
            "Step 2670: Loss = 0.8493\n",
            "Step 2680: Loss = 0.8322\n",
            "Step 2690: Loss = 0.8797\n",
            "Step 2700: Loss = 0.8559\n",
            "Step 2710: Loss = 0.8336\n",
            "Step 2720: Loss = 0.8537\n",
            "Step 2730: Loss = 0.8285\n",
            "Step 2740: Loss = 0.8469\n",
            "Step 2750: Loss = 0.8493\n",
            "Step 2760: Loss = 0.8514\n",
            "Step 2770: Loss = 0.8372\n",
            "Step 2780: Loss = 0.8380\n",
            "Step 2790: Loss = 0.8365\n",
            "Step 2800: Loss = 0.8290\n",
            "Step 2810: Loss = 0.8471\n",
            "Step 2820: Loss = 0.8640\n",
            "Step 2830: Loss = 0.8525\n",
            "Step 2840: Loss = 0.8549\n",
            "Step 2850: Loss = 0.8319\n",
            "Step 2860: Loss = 0.8450\n",
            "Step 2870: Loss = 0.8496\n",
            "Step 2880: Loss = 0.8374\n",
            "Step 2890: Loss = 0.8585\n",
            "Step 2900: Loss = 0.8224\n",
            "Step 2910: Loss = 0.8709\n",
            "Step 2920: Loss = 0.8450\n",
            "Step 2930: Loss = 0.8602\n",
            "Step 2940: Loss = 0.8267\n",
            "Step 2950: Loss = 0.8556\n",
            "Step 2960: Loss = 0.8287\n",
            "Step 2970: Loss = 0.8540\n",
            "Step 2980: Loss = 0.8491\n",
            "Step 2990: Loss = 0.8410\n",
            "Step 3000: Loss = 0.8491\n",
            "Step 3010: Loss = 0.8386\n",
            "Step 3020: Loss = 0.8369\n",
            "Step 3030: Loss = 0.8616\n",
            "Step 3040: Loss = 0.8867\n",
            "Step 3050: Loss = 0.8258\n",
            "Step 3060: Loss = 0.8345\n",
            "Step 3070: Loss = 0.8322\n",
            "Step 3080: Loss = 0.8623\n",
            "Step 3090: Loss = 0.8399\n",
            "Step 3100: Loss = 0.8236\n",
            "Step 3110: Loss = 0.8519\n",
            "Step 3120: Loss = 0.8310\n",
            "Step 3130: Loss = 0.8293\n",
            "Step 3140: Loss = 0.8306\n",
            "Step 3150: Loss = 0.8359\n",
            "Step 3160: Loss = 0.8062\n",
            "Step 3170: Loss = 0.8430\n",
            "Step 3180: Loss = 0.8516\n",
            "Step 3190: Loss = 0.8437\n",
            "Step 3200: Loss = 0.8475\n",
            "Step 3210: Loss = 0.8318\n",
            "Step 3220: Loss = 0.8214\n",
            "Step 3230: Loss = 0.8643\n",
            "Step 3240: Loss = 0.8310\n",
            "Step 3250: Loss = 0.8620\n",
            "Step 3260: Loss = 0.8434\n",
            "Step 3270: Loss = 0.8615\n",
            "Step 3280: Loss = 0.8331\n",
            "Step 3290: Loss = 0.8589\n",
            "Step 3300: Loss = 0.8394\n",
            "Step 3310: Loss = 0.8544\n",
            "Step 3320: Loss = 0.8335\n",
            "Step 3330: Loss = 0.8063\n",
            "Step 3340: Loss = 0.8426\n",
            "Step 3350: Loss = 0.8336\n",
            "Step 3360: Loss = 0.8457\n",
            "Step 3370: Loss = 0.8530\n",
            "Step 3380: Loss = 0.8545\n",
            "Step 3390: Loss = 0.8360\n",
            "Step 3400: Loss = 0.8242\n",
            "Step 3410: Loss = 0.8553\n",
            "Step 3420: Loss = 0.8594\n",
            "Step 3430: Loss = 0.8275\n",
            "Step 3440: Loss = 0.8363\n",
            "Step 3450: Loss = 0.8500\n",
            "Step 3460: Loss = 0.8339\n",
            "Step 3470: Loss = 0.8482\n",
            "Training complete!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "save_directory = \"/Users/premtimsina/Documents/bpbbook/chapter5/pretrained_bert/\"\n",
        "model.save_pretrained(save_directory)\n",
        "tokenizer.save_pretrained(save_directory)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b7KK_QElZXxD",
        "outputId": "77213214-544c-4161-b2a4-dab7c9949ef5"
      },
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "('/Users/premtimsina/Documents/bpbbook/chapter5/pretrained_bert/tokenizer_config.json',\n",
              " '/Users/premtimsina/Documents/bpbbook/chapter5/pretrained_bert/special_tokens_map.json',\n",
              " '/Users/premtimsina/Documents/bpbbook/chapter5/pretrained_bert/vocab.txt',\n",
              " '/Users/premtimsina/Documents/bpbbook/chapter5/pretrained_bert/added_tokens.json')"
            ]
          },
          "metadata": {},
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We conducted further pre-training of bert-based-uncased. Some of the areas for optimization are:\n",
        "\n",
        "1. We used a very simple approach to clean the data, like removing\n",
        "numbers and special characters. It's essential to invest more time in this process and employ more sophisticated techniques.\n",
        "2. We used the simple nltk module to split sentences. While nltk works well for general sentence splitting, clinical notes are written in a more informal manner and often include numbers and stats. As a result, nltk is not the optimal solution. We should use advanced sentence detectors to split sentences.\n",
        "3. The data items we prepared are not entirely accurate. For example, when merging all clinical notes together, the last sentence of clinical note A and the first sentence of clinical note B become sentence A and sentence B, which is not entirely correct.\n",
        "When creating an LLM for your organization, it's crucial to invest a significant amount of time in cleaning the data; otherwise, you'll end up with a suboptimal model despite having a robust model architecture."
      ],
      "metadata": {
        "id": "oIwpXgLGGXxU"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "ytqPVAshGndu"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}